This PR attempts to automate the spam detection process for Job, Event, Codebase and MemberProfile objects using an external LLM service.

LLM Spam Detection Process

1. A SpamModeration record with status SCHEDULED_FOR_CHECK is stored on every Job, Event, Codebase, submission and User (SpamModeration object is attached to the associated MemberProfile) creation.
2. A decoupled external service asuworks/comses.spamcheck queries for these SpamModeration objects (api/spam/get-latest-batch/), analyzes them for spam and submits a spam report to api/spam/update for each one of them.
3. The handler for the api/spam/update on the CoMSES side updates the corresponding SpamModeration object according to the LLM report from the external service.

Starting the LLM Spam Detection Process

The external service asuworks/comses.spamcheck is deployed on an existing JetStream2 instance which is unshelved before the spam check workflow is triggered and shelved automatically after it is done by the following management command:

./manage.py  curator_llm_spam_check

# Following flags are available for troubleshooting purposes:
--skip-changing-instance-state # will skip changing the status of the JetStream2 instance
--skip-shelving-when-done # will start the JetStream2 instance, execute the `CheckSpamWorkflow`, but not shelve it

Environment & Secrets

Following environment variables must be set:

LLM_SPAM_CHECK_API_URL=http://<JetStream2 instance IP>:8001
LLM_SPAM_CHECK_JETSTREAM_SERVER_ID=<JetStream2 instance ID>

JetStream2 Credentials

can be found here: https://js2.jetstream-cloud.org/identity/application_credentials/

secrets/llm_spam_check_jetstream_os_application_credential_secret
secrets/llm_spam_check_jetstream_os_application_credential_id

X-API-Key header for the API

Access to api/spam/update and api/spam/get-latest-batch routes is protected by the X-API-Key header verification.
The key should be set in secrets/llm_spam_check_api_key

ALLOWED_HOSTS

The IP of the JetStream2 instance must be added to Django's ALLOWED_HOSTS